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CLAIMS 

What is claimed is: 

1 . A computer-implemented method of searching hypotheses for 
locations of objects in a playback image corresponding to a recorded image 
generated by a graphical user interface (GUI) of an application program 
comprising: 

capturing the playback image; 

detecting at least one active object in the recorded image; 
searching subsets of hypotheses from the playback image for an object 
according to predetermined criteria; 

* 

recalculating old actions for the object in the playback image by applying 
actions according to an execution scenario and loading a next set of data, when 
the object is found; and , 

checking dynamic conditions. 

2. The method of claim 1 , further comprising repeating the capturing, 
detecting, searching, recalculating, and checking for each of a series of both of 
the playback and recorded images according to the execution scenario. 

3. The method of claim 1, wherein searching comprises performing a first 
search using the predetermined criteria of content and layout. 

4. The method of claim 1, wherein searching comprises performing a first 
, search using the predetermined criteria of size arid distance. 

5. The method of claim 1, wherein searching comprises performing a 
second search using the predetermined criteria of content and sizes of objects. 

6. The method of claim 1, wherein searching comprises performing a 
second search using the predetermined criteria of layout and distances. 
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7. The method of claim 1, wherein searching comprises performing a third 
search using the predetermined criteria of content and sizes of additional objects, 
and layout. 

8. The method of claim 1 , wherein searching comprises performing a third 
search using the predetermined criteria of sizes of active objects and distances. 

9. The method of claim 1, wherein searching comprises performing a third 
search using the predetermined criteria of content for active objects. 

10. An article comprising: a machine accessible medium containing 
instructions, which when executed, result in searching hypotheses for locations 
of objects in a playback image corresponding to a recorded image generated by 
a graphical user interface (GUI) of an application program by 

capturing the playback image; 

detecting at least one active object in the recorded image; 

searching subsets of hypotheses from the playback image for an object 
according to predetermined criteria; 

recalculating old actions for the object in the playback image by applying 
actions according to an execution scenario and loading a next set of data, when 

r 

the object is found; and 

i 

checking dynamic conditions. 

11. The article of claim 10, wherein instructions to search comprise 
instructions to perform a first search using the predetermined criteria of content 
and layout. 

12. The article of claim 10, wherein instructions to search comprise 
instructions to perform a first search using the predetermined criteria of size and 
distance. 
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13. The article of claim 10, wherein instructions to search comprise 
instructions to perform a second search using the predetermined criteria of 
content and sizes of objects. 

14. The article of claim 10, wherein instructions to search comprise 
instructions to perform a second search using the predetermined criteria of layout 
and distances. 

15. The article of claim 10, wherein instructions to search comprise 
instructions to perform a third search using the predetermined criteria of content 
and sizes of additional objects, and layout. 

16. The article of claim 10, wherein instructions to search comprise 
instructions to perform a third search using the predetermined criteria of sizes of 
active objects and distances. 

17. The article of claim 10, wherein instructions tef 'search comprise 
instructions to perform a third search using the predetermined criteria of content 
for active objects. 

18. A cognitive control framework system for automatically controlling 
execution of an application program having a graphical user interface comprising: 

a recording component adapted to capture user input data and images 
displayed by the graphical user interface during a recording phase of execution 
of the application program, and to analyze the captured user input data and 
displayed images to generate an execution scenario during the recording phase, 
and 

a playback component adapted to perform image analysis on images 
displayed by the graphical user interface as a result of processing the simulated 
user input data during the playback phase and captured displayed images from 
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the recording phase, the playback component being adapted to search 
hypotheses for locations of objects in a playback image corresponding to a 
recorded image by 

capturing the playback image; 

detecting at least one active object in the recorded image; 

searching subsets of hypotheses from the playback image for an 
object according to predetermined criteria; 

recalculating old actions for the object in the playback image by 
applying actions according to an execution scenario and loading a next set 
of data, when the object is found; and 

checking dynamic conditions. 

19. The system of claim 18, wherein searching comprises performing a 
first search using the predetermined criteria of content and layout. 

20. The system of claim 18, wherein searching comprises performing a 
first search using the predetermined criteria of size and distance. 

21. The system of claim 18, wherein searching comprises performing a 
second search using the predetermined criteria of content and sizes of objects. 

22. The system of claim 18, wherein searching comprises performing a 
second search using the predetermined criteria of layout and distances. 

23. The system of claim 18, wherein searching comprises performing a 
third search using the predetermined criteria of content and sizes of additional 
objects, and layout. 

24. The system of claim 18, wherein searching comprises performing a 
third search using the predetermined criteria of sizes of active objects and 
distances. 
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25. The system of claim 18, wherein searching comprises performing a 
third search using the predetermined criteria of content for active objects. 
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